Appraising patient preference methods for decision-making in the medical product lifecycle: an empirical comparison.

BACKGROUND: Incorporating patient preference (PP) information into decision-making has become increasingly important to many stakeholders. However, there is little guidance on which patient preference assessment methods, including preference exploration (qualitative) and elicitation (quantitative) methods, are most suitable for decision-making at different stages in the medical product lifecycle (MPLC). This study aimed to use an empirical approach to assess which attributes of PP assessment methods are most important, and to identify which methods are most suitable, for decision-makers' needs during different stages in the MPLC. METHODS: A four-step cumulative approach was taken: 1) Identify important criteria to appraise methods through a Q-methodology exercise, 2) Determine numerical weights to ascertain the relative importance of each criterion through an analytical hierarchy process, 3) Assess the performance of 33 PP methods by applying these weights, consulting international health preference research experts and review of literature, and 4) Compare and rank the methods within taxonomy groups reflecting their similar techniques to identify the most promising methods. RESULTS: The Q-methodology exercise was completed by 54 stakeholders with PP study experience, and the analytical hierarchy process was completed by 85 stakeholders with PP study experience. Additionally, 17 health preference research experts were consulted to assess the performance of the PP methods. Thirteen promising preference exploration and elicitation methods were identified as likely to meet decision-makers' needs. Additionally, eight other methods that decision-makers might consider were identified, although they appeared appropriate only for some stages of the MPLC. CONCLUSIONS: This transparent, weighted approach to the comparison of methods supports decision-makers and researchers in selecting PP methods most appropriate for a given application.

Can Electronic Health Records Databases Complement Spontaneous Reporting System Databases? A Historical-Reconstruction of the Association of Rofecoxib and Acute Myocardial Infarction.

Background: Several initiatives have assessed if mining electronic health records (EHRs) may accelerate the process of drug safety signal detection. In Europe, Exploring and Understanding Adverse Drug Reactions (EU-ADR) Project Focused on utilizing clinical data from EHRs of over 30 million patients from several European countries. Rofecoxib is a prescription COX-2 selective Non-Steroidal Anti-Inflammatory Drugs (NSAID) approved in 1999. In September 2004, the manufacturer withdrew rofecoxib from the market because of safety concerns. In this study, we investigated if the signal concerning rofecoxib and acute myocardial infarction (AMI) could have been identified in EHR database (EU-ADR project) earlier than spontaneous reporting system (SRS), and in advance of rofecoxib withdrawal. Methods: Data from the EU-ADR project and WHO-VigiBase (for SRS) were used for the analysis. Signals were identified when respective statistics exceeded defined thresholds. The SRS analyses was conducted two ways- based on the date the AMI events with rofecoxib as a suspect medication were entered into the database and also the date that the AMI event occurred with exposure to rofecoxib. Results: Within the databases participating in EU-ADR it was possible to identify a strong signal concerning rofecoxib and AMI since Q3 2000 [RR LGPS = 4.5 (95% CI: 2.84-6.72)] and peaked to 4.8 in Q4 2000. In WHO-VigiBase, for AMI term grouping, the EB05 threshold of 2 was crossed in the Q4 2004 (EB05 = 2.94). Since then, the EB05 value increased consistently and peaked in Q3 2006 (EB05 = 48.3) and then again in Q2 2008 (EB05 = 48.5). About 93% (2260 out of 2422) of AMIs reported in WHO-VigiBase database actually occurred prior to the product withdrawal, however, they were reported after the risk minimization/risk communication efforts. Conclusion: In this study, EU-EHR databases were able to detect the AMI signal 4 years prior to the SRS database. We believe that for events that are consistently documented in EHR databases, such as serious events or events requiring in-patient medical intervention or hospitalization, the signal detection exercise in EHR would be beneficial for newly introduced medicinal products on the market, in addition to the SRS data.

Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection.

BACKGROUND: Electronic reporting and processing of suspected adverse drug reactions (ADRs) is increasing and has facilitated automated screening procedures. It is crucial for healthcare professionals to understand the nature and proper use of data available in pharmacovigilance practice. OBJECTIVES: To (a) compare performance of EU-ADR [electronic healthcare record (EHR) exemplar] and FAERS [spontaneous reporting system (SRS) exemplar] databases in detecting signals using "positive" and "negative" drug-event reference sets; and (b) evaluate the impact of timing bias on sensitivity thresholds by comparing all data to data restricted to the time before a warning/regulatory action. METHODS: Ten events with known positive and negative reference sets were selected. Signals were identified when respective statistics exceeded defined thresholds. Main outcome measure Performance metrics, including sensitivity, specificity, positive predictive value and accuracy were calculated. In addition, the effect of regulatory action on the performance of signal detection in each data source was evaluated. RESULTS: The sensitivity for detecting signals in EHR data varied depending on the nature of the adverse events and increased substantially if the analyses were restricted to the period preceding the first regulatory action. Across all events, using data from all years, a sensitivity of 45-73 % was observed for EU-ADR and 77 % for FAERS. The specificity was high and similar for EU-ADR (82-96 %) and FAERS (98 %). EU-ADR data showed range of PPV (78-91 %) and accuracy (78-72 %) and FAERS data yielded a PPV of 97 % with 88 % accuracy. CONCLUSION: Using all cumulative data, signal detection in SRS data achieved higher specificity and sensitivity than EHR data. However, when data were restricted to time prior to a regulatory action, performance characteristics changed in a manner consistent with both the type of data and nature of the ADR. Further research focusing on prospective validation of is necessary to learn more about the performance and utility of these databases in modern pharmacovigilance practice.

Using real-world healthcare data for pharmacovigilance signal detection - the experience of the EU-ADR project.

A prospective pharmacovigilance signal detection study, comparing the real-world healthcare data (EU-ADR) and two spontaneous reporting system (SRS) databases, US FDA's Adverse Event Reporting System and WHO's Vigibase is reported. The study compared drug safety signals found in the EU-ADR and SRS databases. The potential for signal detection in the EU-ADR system was found to be dependent on frequency of the event and utilization of drugs in the general population. The EU-ADR system may have a greater potential for detecting signals for events occurring at higher frequency in general population and those that are commonly not considered as potentially a drug-induced event. Factors influencing various differences between the datasets are discussed along with potential limitations and applications to pharmacovigilance practice.

Postmarketing safety surveillance : where does signal detection using electronic healthcare records fit into the big picture?

The safety profile of a drug evolves over its lifetime on the market; there are bound to be changes in the circumstances of a drug's clinical use which may give rise to previously unobserved adverse effects, hence necessitating surveillance postmarketing. Postmarketing surveillance has traditionally been carried out by systematic manual review of spontaneous reports of adverse drug reactions. Vast improvements in computing capabilities have provided opportunities to automate signal detection, and several worldwide initiatives are exploring new approaches to facilitate earlier detection, primarily through mining of routinely-collected data from electronic healthcare records (EHR). This paper provides an overview of ongoing initiatives exploring data from EHR for signal detection vis-à-vis established spontaneous reporting systems (SRS). We describe the role SRS has played in regulatory decision making with respect to safety issues, and evaluate the potential added value of EHR-based signal detection systems to the current practice of drug surveillance. Safety signal detection is both an iterative and dynamic process. It is in the best interest of public health to integrate and understand evidence from all possibly relevant information sources on drug safety. Proper evaluation and communication of potential signals identified remains an imperative and should accompany any signal detection activity.

A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases.

BACKGROUND: The growing interest in using electronic healthcare record (EHR) databases for drug safety surveillance has spurred development of new methodologies for signal detection. Although several drugs have been withdrawn postmarketing by regulatory authorities after scientific evaluation of harms and benefits, there is no definitive list of confirmed signals (i.e. list of all known adverse reactions and which drugs can cause them). As there is no true gold standard, prospective evaluation of signal detection methods remains a challenge. OBJECTIVE: Within the context of methods development and evaluation in the EU-ADR Project (Exploring and Understanding Adverse Drug Reactions by integrative mining of clinical records and biomedical knowledge), we propose a surrogate reference standard of drug-adverse event associations based on existing scientific literature and expert opinion. METHODS: The reference standard was constructed for ten top-ranked events judged as important in pharmacovigilance. A stepwise approach was employed to identify which, among a list of drug-event associations, are well recognized (known positive associations) or highly unlikely ('negative controls') based on MEDLINE-indexed publications, drug product labels, spontaneous reports made to the WHO's pharmacovigilance database, and expert opinion. Only drugs with adequate exposure in the EU-ADR database network (comprising ≈60 million person-years of healthcare data) to allow detection of an association were considered. Manual verification of positive associations and negative controls was independently performed by two experts proficient in clinical medicine, pharmacoepidemiology and pharmacovigilance. A third expert adjudicated equivocal cases and arbitrated any disagreement between evaluators. RESULTS: Overall, 94 drug-event associations comprised the reference standard, which included 44 positive associations and 50 negative controls for the ten events of interest: bullous eruptions; acute renal failure; anaphylactic shock; acute myocardial infarction; rhabdomyolysis; aplastic anaemia/pancytopenia; neutropenia/agranulocytosis; cardiac valve fibrosis; acute liver injury; and upper gastrointestinal bleeding. For cardiac valve fibrosis, there was no drug with adequate exposure in the database network that satisfied the criteria for a positive association. CONCLUSION: A strategy for the construction of a reference standard to evaluate signal detection methods that use EHR has been proposed. The resulting reference standard is by no means definitive, however, and should be seen as dynamic. As knowledge on drug safety evolves over time and new issues in drug safety arise, this reference standard can be re-evaluated.

EU-ADR healthcare database network vs. spontaneous reporting system database: preliminary comparison of signal detection.

The EU-ADR project aims to exploit different European electronic healthcare records (EHR) databases for drug safety signal detection. In this paper we report the preliminary results concerning the comparison of signal detection between EU-ADR network and two spontaneous reporting databases, the Food and Drug Administration and World Health Organization databases. EU-ADR data sources consist of eight databases in four countries (Denmark, Italy, Netherlands, and United Kingdom) that are virtually linked through distributed data network. A custom-built software (Jerboa©) elaborates harmonized input data that are produced locally and generates aggregated data which are then stored in a central repository. Those data are subsequently analyzed through different statistics (i.e. Longitudinal Gamma Poisson Shrinker). As potential signals, all the drugs that are associated to six events of interest (bullous eruptions - BE, acute renal failure - ARF, acute myocardial infarction - AMI, anaphylactic shock - AS, rhabdomyolysis - RHABD, and upper gastrointestinal bleeding - UGIB) have been detected via different data mining techniques in the two systems. Subsequently a comparison concerning the number of drugs that could be investigated and the potential signals detected for each event in the spontaneous reporting systems (SRSs) and EU-ADR network was made. SRSs could explore, as potential signals, a larger number of drugs for the six events, in comparison to EU-ADR (range: 630-3,393 vs. 87-856), particularly for those events commonly thought to be potentially drug-induced (i.e. BE: 3,393 vs. 228). The highest proportion of signals detected in SRSs was found for BE, ARF and AS, while for ARF, and UGIB in EU-ADR. In conclusion, it seems that EU-ADR longitudinal database network may complement traditional spontaneous reporting system for signal detection, especially for those adverse events that are frequent in general population and are not commonly thought to be drug-induced. The methodology for signal detection in EU-ADR is still under development and testing phase.

Perspectives on the use of data mining in pharmaco-vigilance.

In the last 5 years, regulatory agencies and drug monitoring centres have been developing computerised data-mining methods to better identify reporting relationships in spontaneous reporting databases that could signal possible adverse drug reactions. At present, there are no guidelines or standards for the use of these methods in routine pharmaco-vigilance. In 2003, a group of statisticians, pharmaco-epidemiologists and pharmaco-vigilance professionals from the pharmaceutical industry and the US FDA formed the Pharmaceutical Research and Manufacturers of America-FDA Collaborative Working Group on Safety Evaluation Tools to review best practices for the use of these methods.In this paper, we provide an overview of: (i) the statistical and operational attributes of several currently used methods and their strengths and limitations; (ii) information about the characteristics of various postmarketing safety databases with which these tools can be deployed; (iii) analytical considerations for using safety data-mining methods and interpreting the results; and (iv) points to consider in integration of safety data mining with traditional pharmaco-vigilance methods. Perspectives from both the FDA and the industry are provided. Data mining is a potentially useful adjunct to traditional pharmaco-vigilance methods. The results of data mining should be viewed as hypothesis generating and should be evaluated in the context of other relevant data. The availability of a publicly accessible global safety database, which is updated on a frequent basis, would further enhance detection and communication about safety issues.

Data mining in pharmacovigilance: the need for a balanced perspective.

Data mining is receiving considerable attention as a tool for pharmacovigilance and is generating many perspectives on its uses. This paper presents four concepts that have appeared in various professional venues and represent potential sources of misunderstanding and/or entail extended discussions: (i) data mining algorithms are unvalidated; (ii) data mining algorithms allow data miners to objectively screen spontaneous report data; (iii) mathematically more complex Bayesian algorithms are superior to frequentist algorithms; and (iv) data mining algorithms are not just for hypothesis generation. Key points for a balanced perspective are that: (i) validation exercises have been done but lack a gold standard for comparison and are complicated by numerous nuances and pitfalls in the deployment of data mining algorithms. Their performance is likely to be highly situation dependent; (ii) the subjective nature of data mining is often underappreciated; (iii) simpler data mining models can be supplemented with 'clinical shrinkage', preserving sensitivity; and (iv) applications of data mining beyond hypothesis generation are risky, given the limitations of the data. These extended applications tend to 'creep', not pounce, into the public domain, leading to potential overconfidence in their results. Most importantly, in the enthusiasm generated by the promise of data mining tools, users must keep in mind the limitations of the data and the importance of clinical judgment and context, regardless of statistical arithmetic. In conclusion, we agree that contemporary data mining algorithms are promising additions to the pharmacovigilance toolkit, but the level of verification required should be commensurate with the nature and extent of the claimed applications.